Automatic email classification

The endlessly increasing volume of unsolicited emails (a.k.a. spam) has become more and more of a concern. Its hassles range from a daily loss of time for the end-user, required to keep her mailbox clean, to a financial loss for the ISPs, constantly in need of larger bandwidths and disk space. According to a recent study, MSN and AOL discard together almost five billion of such emails every day[2]. Several technical solutions have been proposed to counter this plague, which are mainly deterministic. The most used are black lists, white lists and score-based filters[7, 3]. Black lists use a large database of email servers known to be owned by spammers to block any email coming from them. White lists allow only emails from known senders to go through. Scorebased filters give a number of points to each incoming emails according to their contents, e.g. 5 points when the word “free” is present. Emails with high scores are then blocked. However the efficiency of such techniques is limited and spammers have already found ways of bypassing them. A few years ago a new solution has been proposed: bayesian classification[3, 4, 6, 5]. The results showed in these papers and the number of related software currently under development[7] seem to indicate a large potential. This is what this project will investigate. I propose to tackle this issue with two improvements in mind. Firstly, the adaptation of the boosting chain procedure used by some face detection algorithms[8]. This should improve the speed of the overall procedure by adopting a simple-to-complex strategy, complex classifiers being applied only to emails not classified by simpler ones. At the same time it might decrease the false positive rate. Secondly, the creation of a language-independent classifier. Being from France, I receive emails both in French and in English. Using a unique classifier on all emails seems quite inefficient considering the vastly different statistics of words and grammatical rules in both languages. The language-independence will be handled in the most general way to allow the classifier to deal with any language. However, the lack of data and personal knowledge will restrict the study of the generalization to Polish. If these two improvements prove to be successful, a third one will be handled: the generalization of the classifier to n classes. Current classifiers have only two classes, namely “spam” and “not spam”. A more interesting algorithm would also handle classes like “friends”, “family”, “work”, etc. thus leading to a true automatic email classifier.